Intrinsic Motivation and Mental Replay enable Efficient Online Adaptation in Stochastic Recurrent Networks

Autonomous robots need to interact with unknown, unstructured and changing environments, constantly facing novel challenges. Therefore, continuous online adaptation for lifelong-learning and the need of sample-efficient mechanisms to adapt to changes in the environment, the constraints, the tasks, or the robot itself are crucial. In this work, we propose a novel framework for probabilistic online motion planning with online adaptation based on a bio-inspired stochastic recurrent neural network. By using learning signals which mimic the intrinsic motivation signalcognitive dissonance in addition with a mental replay strategy to intensify experiences, the stochastic recurrent network can learn from few physical interactions and adapts to novel environments in seconds. We evaluate our online planning and adaptation framework on an anthropomorphic KUKA LWR arm. The rapid online adaptation is shown by learning unknown workspace constraints sample-efficiently from few physical interactions while following given way points.Comment: accepted in Neural Network

“I will survive”: online streaming and the chart survival of music tracks

Digital streaming has had a profound effect on the commercial music sector and now accounts for 80% of industry revenues in the United States. This study investigates the consumption of music on digital streaming platforms by analyzing the factors affecting the chart survival of individual music tracks. Our data are taken from the Spotify Global Top 200 between January 2017 and January 2020, containing observations on 3,007 unique tracks by 642 artists over 1,087 days. We identify a number of unique consumption traits applicable to online streaming services, which we use to explain variations in chart longevity. We find a positive association between the amount of time a track spends in the chart and the involvement of a major label. We also find that the level of competition from other chart entries, as well as some elements related to the pattern of diffusion, associates significantly with the likelihood of chart survival. The study highlights several important managerial implications for key industry stakeholders

Evolutionary training and abstraction yields algorithmic generalization of neural computers

A key feature of intelligent behaviour is the ability to learn abstract strategies that scale and transfer to unfamiliar problems. An abstract strategy solves every sample from a problem class, no matter its representation or complexity—similar to algorithms in computer science. Neural networks are powerful models for processing sensory data, discovering hidden patterns and learning complex functions, but they struggle to learn such iterative, sequential or hierarchical algorithmic strategies. Extending neural networks with external memories has increased their capacities to learn such strategies, but they are still prone to data variations, struggle to learn scalable and transferable solutions, and require massive training data. We present the neural Harvard computer, a memory-augmented network-based architecture that employs abstraction by decoupling algorithmic operations from data manipulations, realized by splitting the information flow and separated modules. This abstraction mechanism and evolutionary training enable the learning of robust and scalable algorithmic solutions. On a diverse set of 11 algorithms with varying complexities, we show that the neural Harvard computer reliably learns algorithmic solutions with strong generalization and abstraction, achieves perfect generalization and scaling to arbitrary task configurations and complexities far beyond seen during training, and independence of the data representation and the task domain

Online Learning with Stochastic Recurrent Neural Networks using Intrinsic Motivation Signals

Continuous online adaptation is an essential ability for the vision of fully autonomous and lifelong-learning robots. Robots need to be able to adapt to changing environments and constraints while this adaption should be performed without interrupting the robot’s motion. In this paper, we introduce a framework for probabilistic online motion planning and learning based on a bio-inspired stochastic recurrent neural network. Furthermore, we show that the model can adapt online and sample-efficiently using intrinsic motivation signals and a mental replay strategy. This fast adaptation behavior allows the robot to learn from only a small number of physical interactions and is a promising feature for reusing the model in different environments. We evaluate the online planning with a realistic dynamic simulation of the KUKA LWR robotic arm. The efficient online adaptation is shown in simulation by learning an unknown workspace constraint using mental replay and cognitive dissonance as intrinsic motivation signal

Intrinsic motivation and mental replay enable efficient online adaptation in stochastic recurrent networks

Autonomous robots need to interact with unknown, unstructured and changing environments, constantly facing novel challenges. Therefore, continuous online adaptation for lifelong-learning and the need of sample-efficient mechanisms to adapt to changes in the environment, the constraints, the tasks, or the robot itself are crucial. In this work, we propose a novel framework for probabilistic online motion planning with online adaptation based on a bio-inspired stochastic recurrent neural network. By using learning signals which mimic the intrinsic motivation signal cognitive dissonance in addition with a mental replay strategy to intensify experiences, the stochastic recurrent network can learn from few physical interactions and adapts to novel environments in seconds. We evaluate our online planning and adaptation framework on an anthropomorphic KUKA LWR arm. The rapid online adaptation is shown by learning unknown workspace constraints sample-efficiently from few physical interactions while following given way points

SKID RAW: Skill Discovery From Raw Trajectories

Integrating robots in complex everyday environments requires a multitude of problems to be solved. One crucial feature among those is to equip robots with a mechanism for teaching them a new task in an easy and natural way. When teaching tasks that involve sequences of different skills, with varying order and number of these skills, it is desirable to only demonstrate full task executions instead of all individual skills. For this purpose, we propose a novel approach that simultaneously learns to segment trajectories into reoccurring patterns and the skills to reconstruct these patterns from unlabelled demonstrations without further supervision. Moreover, the approach learns a skill conditioning that can be used to understand possible sequences of skills, a practical mechanism to be used in, for example, human-robot-interactions for a more intelligent and adaptive robot behaviour. The Bayesian and variational inference based approach is evaluated on synthetic and real human demonstrations with varying complexities and dimensionality, showing the successful learning of segmentations and skill libraries from unlabelled data

Understand-Compute-Adapt: Neural Networks for Intelligent Agents

An artificial intelligent agent needs to be equipped with a multitude of abilities in order to interact in the world among us. These requirements for intelligent behaviour can roughly be separated into two main categories, cognitive abilities and physical skills. The cognitive abilities refer to cognition and problem solving, whereas the physical skills correspond to movements of an intelligent robot in the real world. In this thesis, we investigate three research questions tackling those different abilities. Precisely, how can new knowledge be taught to a robot in a natural way? How can neural networks learn abstract solution strategies that are independent of the task complexity, data representation and task domain? How can a robot efficiently adapt its movement during execution with a bio-inspired stochastic neural network? These questions span core requirements for intelligent autonomous agents, which we categorize as Understand-Compute-Adapt (UCA), in the style of the classical Sense-Plan-Act framework in robotics. To answer these questions, we investigate neural network based models on these cognitive and physical abilities. In detail, the first question tackles the ability of cognition, which refers to an understanding of the world and is investigated by learning a set of skills from unlabelled demonstrations of full task executions. Therefore, we studied the task of trajectory segmentation and skill library learning. To provide a natural interface for teaching a robot new tasks, it is desirable to have the user only demonstrating the desired task, without worrying about all the skills that are required for the task and without manually annotating the demonstrations. Such an interface not only enables non-experts to teach robots, but also provides a cheaper approach to teaching robots, as demonstrating all individual skills or segmenting and labelling demonstrations by hand is time consuming and expensive. The approach proposed here learns to segment trajectories and the required skill library simultaneously from unlabelled demonstrations. In addition to this segmenting and skill discovery, the approach also learns the relations between individual skills, i.e., modelling how likely a certain skill follows after another skill. This additional knowledge, or understanding, can be used, for example, in human-robot-interaction scenarios by predicting the human behaviour and therefore enables a more intelligent adaptive behaviour of the robot. The approach was successfully evaluated on multiple different trajectory datasets with varying complexities. The second aforementioned required cognitive ability, problem solving, refers to the second question and the Compute step. In particular, we investigated the challenge of learning algorithmic solutions, i.e., learning abstract strategies that can easily be transferred to unfamiliar problem instantiations. This transferring of abstract knowledge and solution strategies into novel domains is another crucial feature of intelligent behaviour. Therefore, we investigated the learning of algorithmic solutions that are characterized by three requirements highlighting the abstract nature of the solution: scaling to arbitrary task configurations and complexities, and the independence of both the data representations as well as the task domain. For this purpose we developed a novel framework, the Neural Harvard Computer, that is based on memory-augmented neural networks and whose modular design is inspired by the von Neumann and Harvard architectures of modern computers. This framework enables the learning of abstract algorithmic solutions through its modular design and the separation of information flow into data and control signals. The algorithmic solution is learned in a reinforcement learning setting and solely operating on the control signal flow, enabling the independence of the data representation and task domain. We evaluated the framework's generalization and abstraction features by learning 11 different algorithms, where the approach was able to reliably learn algorithmic solutions with perfect generalization and abstraction, allowing to solve problems with complexities far beyond seen during training and by straight forward transfer to novel task representations and domains. Ultimately an intelligent robot has to interact in the real world, giving rise to the third entry Adapt, the question of efficient online adaptation. In order to cope with the complex, dynamic and often unstructured real world, in addition to dealing with other agents and humans, the agent has to be able to adapt its models and movements while interacting. This online adaptation belongs to the mentioned physical skills that are required for intelligent behaviour. Moreover, this online adaptation has to be efficient in terms of number of physical interactions and be task-independent, as not every situation can be foreseen when constructing the agent or the method. In this thesis, we studied online adaptation within a bio-inspired spiking neural network that generates movements by simulating its inherent dynamics. The underlying stochastically spiking neurons mimic the behaviour of hippocampal place cells and their decoded activity represents the planned movement. Task-independent adaptation is achieved by using intrinsic motivation signals inspired by cognitive dissonance to guide the learning. These signals capture the discrepancy between the agents expectation of the world (the current model) and the observations of the world, and the online adaptation is triggered and steered through this mismatch. Sample-efficiency is accomplished by using a mental replay strategy to intensify experienced situations and is implemented by using the inherent stochasticity of the framework. We evaluated this framework for online model adaptation and movement generation on an anthropomorphic KUKA LWR arm, where the robot has to adapt to unknown obstacles while performing a waypoint following task. The online adaptation happens within seconds and from few physical interactions while keeping interacting with the environment. In summary, this thesis investigates three key aspects of intelligent behaviour with respect to cognitive and physical abilities. In more detail, we investigated how neural network based models can be used from learning to understand over learning to compute to learning to adapt to tackle the three raised research question. Each topic has its own requirements on the used neural network model and the learning mechanism. This modularity and diversity of subroutines is a crucial aspect for creating artificial intelligence

What containment strategy leads us through the pandemic crisis? An empirical analysis of the measures against the COVID-19 pandemic

Since January 2020, the COVID-19 outbreak has been progressing at a rapid pace. To keep the pandemic at bay, countries have implemented various measures to interrupt the transmission of the virus from person to person and prevent an overload of their health systems. We analyze the impact of these measures implemented against the COVID-19 pandemic by using a sample of 68 countries, Puerto Rico and the 50 federal states of the United States of America, four federal states of Australia, and eight federal states of Canada, involving 6,941 daily observations. We show that measures are essential for containing the spread of the COVID-19 pandemic. After controlling for daily COVID-19 tests, we find evidence to suggest that school closures, shut-downs of non-essential business, mass gathering bans, travel restrictions in and out of risk areas, national border closures and/or complete entry bans, and nationwide curfews decrease the growth rate of the coronavirus and thus the peak of daily confirmed cases. We also find evidence to suggest that combinations of these measures decrease the daily growth rate at a level outweighing that of individual measures. Consequently, and despite extensive vaccinations, we contend that the implemented measures help contain the spread of the COVID-19 pandemic and ease the overstressed capacity of the healthcare systems